Optimized job scheduling and execution in a distributed computing grid

ABSTRACT

A disclosed example involves determining whether there is at least one valid combination of nodes and links from the network of nodes with capability and capacity over time to complete a computer-executable job by a deadline. A total cost combination of nodes and links is selected from among the at least one valid combination of nodes and links with the capability and capacity over time to complete the computer-executable job by the deadline. The computer-executable job is scheduled to be executed on at least one selected node. The scheduling is based on compiled instructions comprising the computer-executable job. At least some of the link capacity of at least one of the links connected to the at least one selected node is reserved, to match a job transport capacity requirement of the computer-executable job.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to data processing in general, and specificallyto arrangements for optimizing job scheduling and execution in adistributed computing grid.

2. Related Art

In traditional data processing environments, a set of servers (i.e.,computers, such as mainframes, midrange processors, blade servers, andthe like) interact with storage (such as disk, tape, or network-attachedstorage) on a dedicated basis to process compute jobs such as payroll,e-commerce, billing, and so forth. Additional elements, such asfirewalls, load balancers, Local Area Networks, Storage Area Networks,and the like, are also typically engaged. For example, servers A, B, andC may be dedicated to a payroll application, servers D, E, and F may bededicated to customer technical support, and servers G, H, and I may bededicated to web applications.

A limitation of this conventional approach is that the capacitydedicated to individual applications may be too little or too great atany given time. If the capacity is too great, it means that the owner ofthis infrastructure may have overpaid for the hardware and softwarecomprising the data processing environment. If it is too little, itmeans that the application may not meet performance objectives such asthe number of simultaneous users supported, throughput, response timeand latency, or the like.

An emerging approach is called “grid computing.” Grid computingtypically involves a number of geographically dispersed compute nodes.If an application needs to be run, and capacity of the appropriate typeand configuration is available at one of the nodes, the job is scheduledto run at that node. A problem with this conventional job schedulingapproach is that it ignores network considerations, except for theavailability of a basic connectivity path to the node.

Consequently, the inventor has realized that, even though it may be truethat a node has available capacity, it may not be the best node for thejob, due to the total costs involved in moving the application and thedata required for the application, and additional data, packets, ortransactions as the job runs. These costs include the cost of transport,as well as the cost of delays due to bandwidth that may be insufficientto move the data to the node on a timely basis. These delay costs can bequantified due to user dissatisfaction, regulatory requirements andfinancial penalties, competitive needs, job deadlines, and so forth.

Moreover, conventional grid environments typically provision largefixed-bandwidth connections between nodes, for example, several GigabitEthernet or even several 10 Gigabit Ethernet connections are used in theTeraGrid backbone sponsored by the National Science Foundation. Much ofthe time, most of this capacity is unused, and consequently, users orfirms may overpay for unused capacity.

It would be economical to utilize a switched line and pay only forneeded service, rather than lease an expensive dedicated but underusedline. In this regard, emerging technologies permit bandwidth to beallocated “on demand” on a link or end-to-end basis. Bandwidth on demand(BoD) is sometimes called (or is closely related to) dynamic bandwidthallocation, load balancing, committed information rates, rate shaping,quality of service (QoS) management, traffic management, trafficengineering, bandwidth minimums, bandwidth maximums, and the like. Asgenerally understood and broadly used here, BoD temporarily flexiblyprovides capacity on a link to accommodate changes in the volume (e.g.,packets or megabits per second) or characteristics (e.g., jitter, packetloss) of demand, the capacity being dynamically increased or decreasedas specified through a control interface. A typical BoD implementationinvolves a router (or other network element as a switch, opticaladd/drop multiplexer, and the like) with the capability to perform thebandwidth allocation. Such routers can also be directed to establish orotherwise support the establishment of logical links on demand toprovide more capacity (subject to the ultimate physical capacity of alink), and then be directed to dissolve the link as the traffic demandwithdraws. The network element is typically coupled with a higher levelentity, such as a software policy management layer, that tells thenetwork element what to do. Various ways are known in the art toimplement BoD, but none appear to be linked to or combined withscheduling jobs on nodes on the network.

What is needed in the art is way for a grid computing environment jobscheduler to synergetically interoperate with such network functionalityto optimize the overall performance and cost of distributed computing.

SUMMARY

The invention provides a system for scheduling at least one job forexecution on a network of nodes joined by links. The system has (a) ajob planner including (a1) a validity portion configured to determinethe nodes' ability to run the one or more jobs at a given time and (a2)a cost portion configured to determine costs to run the at least one jobat respective nodes at a given time. The system also has (b) a jobscheduler configured to cooperate with the validity portion and the costportion so as to determine at least one optimum node that is able to runthe at least one job and that has a lowest cost.

Additionally, the invention provides a method for scheduling at leastone job for execution on at least one optimum node in a network of nodesconnected by links. The method involves determining which nodes arecapable of running the at least one job, analyzing measures of totalcost associated with execution of the at least one job on respectivenodes that were determined to be capable of running the at least onejob, and based on a measure of least total cost, selecting the at leastone optimum node for running the at least one job.

Further, an arrangement combining job scheduling with bandwidth ondemand (BoD) involves a system for scheduling at least one job forexecution on a network of nodes joined by links having respective linkcapacities, each job associated with a transport capacity requirement.The system has a job scheduler configured to schedule the at least onejob to be executed on at least one selected node, and a link managerconfigured to reserve at least some of the link capacity of at least oneof the links connected to the at least one selected node, to match thejob transport capacity requirement.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the described embodiments is betterunderstood by reference to the following Detailed Description consideredin connection with the accompanying drawings, in which like referencenumerals refer to identical or corresponding parts throughout, and inwhich:

FIGS. 1A and 1B (which may collectively be referred to as “FIG. 1”)respectively illustrate an embodiment of a job scheduling and executionsystem and a network of nodes and links; and

FIG. 2 illustrates an embodiment of a job scheduling and executionmethod.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the inventionis not intended to be limited to the specific terminology so selected,and it is to be understood that each specific element includes alltechnical equivalents that operate in a similar manner to accomplish asimilar purpose. Various terms that are used in this specification areto be given their broadest reasonable interpretation when used tointerpret the claims.

Moreover, features and procedures whose implementations are well knownto those skilled in the art are omitted for brevity. For example,initiation and termination of loops, and the corresponding incrementingand testing of loop variables, may be only briefly mentioned orillustrated, their details being easily surmised by skilled artisans.Thus, the steps involved in methods described herein may be readilyimplemented by those skilled in the art without undue experimentation.

Further, various aspects, features and embodiments may be described as aprocess that can be depicted as a flowchart, a flow diagram, a structurediagram, or a block diagram. Although a flowchart may describe theoperations as a sequential process, many of the operations can beperformed in parallel, concurrently, or in a different order than thatdescribed. Operations not needed or desired for a particularimplementation may be omitted. A process or steps thereof may correspondto a method, a function, a procedure, a subroutine, a subprogram, and soforth, or any combination thereof.

As background, various terms and concepts are explained, with theunderstanding that the scope of the claims should not be limited byparticular explanations or embodiments. A network includes a group ofnodes with potentially varied capabilities connected by links. Each nodehas specific fixed capacity and capabilities, e.g., 12 servers with theability to run a particular operating system, 3.7 Terabytes ofenterprise storage, 2 routers with a Terabit of bandwidth each, and soforth. Each link has specific capabilities and capacity, e.g., four 10Gigabit Ethernet service channels.

Each link may have capacity that may be subdivided into logical linkswith corresponding bandwidth allocations for different purposes. Forexample, a physical capacity of 10 gigabits per second may be allocatedas 10 gigabits to customer or application A, or possibly 2 gigabits persecond to customer A and 8 gigabits per second to customer B, or 1gigabit per second to each of 10 customers or applications, and soforth. These logical or virtual link bandwidth amounts may be adjustedupward or downward, with such allocations occurring dynamically overtime periods of weeks, days, hours, minutes, or even seconds. When a jobis to be run, the method described below determines an optimal validnode, reserves link and node capabilities at a given capacity, andschedules and executes the job at the optimal valid node.

In general, we use terms such as “job,” “node,” and “run” very generallyand expansively. For example, a job may be a processor intensive, dataintensive activity, such as seismic analysis for petroleum exploration,which involves large data sets requiring substantial storage, and highspeed CPUs. Another job may be a processor-intensive activity that doesnot require any data storage, such as testing a large number to see ifit is prime. Or it may be a data intensive activity that does notrequire any processors, such as storing a large data set for archivalpurposes for seven years on tape or content-addressed storage. Nodes mayhave general purpose capability, including processing on a variety ofcomputing platforms, operating system, and middleware configurations,disk storage, tape storage, and the like, or may have somewhat limitedcapabilities, for example, a node may not have capability to process orstore data, but may be a network hub that switches and routes trafficdestined for other nodes. Thus, we use the term “run” to mean usingresources at a node over time consonant with the objectives of the job.Consequently, storing data in a tape silo is considered to be an exampleof “running” a job, the same way as “running” an application byexecuting it in a processing environment.

In the method described herein, first the requirements of the job may bedetermined relative to the capabilities of each node. In practice, e.g.,a specific job may require the Solaris 8.0 operating system and aSunFire 15K computer, available from Sun Microsystems of Menlo Park,Calif., but a particular node may only have a configuration andcapability available to run the MVS operating system on a zSeriescomputer available from International Business Machines of Armonk, N.Y.Even if a SunFire 15K were physically present, it may not have capacityavailable within a reasonable planning/scheduling window. A set ofcandidate nodes may be determined by examining all nodes in the system,and evaluating whether they have the capability and the capacity overtime to run the job. By “over time,” we mean examining whether thatcapacity will continue to exist for the expected duration of the job,based on a schedule of jobs starting and completing and thus acquiringand releasing capacity.

Once a set of candidate nodes have been determined, for each candidatenode, it is determined whether there is the right link capability andcapacity available, were the compute job to be run at that candidatenode. Capability and capacity may include specific protocols, quality ofservice, bandwidth, and the like. Link bandwidth is required for avariety of reasons. Data needed to run the job may be in one or morelocations in the system, and at the beginning of the job, or as the jobproceeds, data sets may need to be transferred or accessed. Activeend-users may exist either at various nodes or may access the systemthrough various nodes. Some of the candidate nodes may be ruled out atthis stage, due to a lack of link capability and capacity. Thisdetermines a set of valid nodes. Due to capabilities required andcapacity needed, there may not be any candidate nodes; and even if thereare, there may not be any valid nodes.

If there are not any valid nodes, the method stops. If there are validnodes, the method continues, finding an optimal valid node. Such anoptimal valid node (there may be more than one) is selected based onpicking one with the lowest total cost. Such a cost may be determinedbased on one or more factors, including:

-   -   the cost of the CPU,    -   the cost of the storage,    -   the cost of the network,    -   the cost of delaying the job, and/or    -   other factors not specifically listed here.

Once an optimal valid node is selected, the compute, storage, network,and/or other capacity are reserved for the duration of the job inaccordance with the requirements of the job, and the job is executed atthe selected optimal valid node.

One embodiment of a system involves several elements operating withrespect to a network of nodes connected by links, shown schematically inFIGS. 1A and 1B:

-   -   Nodes 110A through 110D, which are physical entities at specific        geographic locations comprising zero or more components such as        processors, storage, security elements, and network elements        such as routers, switches, gateways, etc.    -   Links 120A through 120E, which are physical entities connecting        pairs of nodes, made up of zero or more service channels of        various capacity supporting one or more network protocols, such        as TCP/IP, Fibre Channel, ESCON, and the like.    -   A node capability database 132, maintaining information,        including capacity, about the number and type of components at        each node.    -   A node utilization table 134, showing planned utilization over        time of each component at each node in accordance with a job        schedule.    -   A node cost table 136, maintaining information on the cost (or        price) to use each component at each node over time.        Equivalently, the node cost table may be viewed from a        service-oriented perspective where rather than providing static        cost structure, dynamic pricing information may be presented.    -   A link capability database 142, maintaining information,        including capacity, about the number and type of service        channels in each link.    -   A link utilization table 144, showing planned utilization over        time of each service channel of each link in accordance with a        job schedule.    -   A link cost table 146, maintaining information on the cost (or        price) to use each service channel on each link over time.        Equivalently, the link cost table may be viewed from a        service-oriented perspective where rather than providing static        cost structure, dynamic pricing information may be presented or        accessed.    -   A node manager 130, which provides access to information        contained in the node capability database, the node utilization        table, and the node cost table, and allows new capability        utilization plans to be scheduled and overlaid on the existing        node utilization plan contained in the node utilization table.    -   A link manager 140, which provides access to information        contained in the link capability database, the link utilization        table, and the link cost table, and allows new capability        utilization plans to be scheduled and overlaid via updating the        link utilization table.    -   A job schedule 152, which has a list of each job to be run, and        the planned nodes and links to be (at least partially) utilized        by each job over time.    -   Job metadata 162, which includes all relevant information        concerning the job, such as the environment needed for running        the job, priority, required start, planned utilization of        resources over time, required latencies.    -   A job planner 160, which, given a new job to be run and its        associated requirements, determines whether there are any nodes        in the system that can run it, and if so, the lowest total cost        node and associated links.    -   A job submission module 164, that accepts jobs to be submitted,        and either reports back that they cannot be run due to lack of        resources or reports their status and or completion.    -   A job scheduler 150, which then adds the job to the job        schedule, updates the node utilization table and link        utilization table, and causes the job to execute at the correct        time at the planned node.

Briefly, a group of physical sites (and intermediate network nodes) areconnected by network links. A link manager 140 monitors status of thenetwork, maintains inventory concerning links and topology information,and can dynamically allocate bandwidth and associated quality of serviceto specific purposes subject to the ultimate physical limits of eachconnection, which are based on capacity of the underlying medium and thespecific optoelectronics in the network elements provisioned into thenetwork. The link manager 140 can also maintain and communicate costinformation regarding each link, which may be somewhat static, e.g., somany dollars per gigabit of bandwidth from point a to point b, or may bedynamic, e.g., $1.23 per gigabit of bandwidth this second, and $1.76 pergigabit of bandwidth this next second, and $1.54 per gigabit ofbandwidth this next second, and so on second by second, minute byminute.

Job scheduler 150 maintains a list of jobs to be run. Typically, thislist changes, as jobs complete, or new jobs to be run are added. When ajob is to be run, the job scheduler 150 acquires or determinesinformation concerning:

-   -   the priority of the job,    -   the CPU and storage requirements for the job,    -   specific environment requirements (e.g., operating system),    -   network access requirements, and/or    -   other considerations not specifically listed.

The job scheduler then may use any of a variety of algorithms orheuristics to determine the order in which jobs will be run, such as,for example:

-   -   “First Come, First Serve,”    -   “Shortest Job First,”    -   “First Fit,”    -   priority queuing, and/or    -   other approaches not specifically listed.

When a job is to be run, a location is determined that meetsrequirements such as:

-   -   available capacity,    -   specific environment, and/or    -   other requirements not specifically listed.

If more than one location will serve, the location is selected based onminimizing a total cost function. The total cost function may be basedon one or more considerations such as:

-   -   costs for servers,    -   costs for CPU capacity,    -   costs for storage,    -   costs for security elements such as firewalls,    -   costs for network bandwidth,    -   delay due to node availability or latency, and/or    -   other costs not specifically listed.

With this background in mind, reference is now made to FIG. 1 for a moredetailed explanation.

System 100 operates with respect to nodes 110 x, shown here as nodes110A, 110B, 110C, and 110D, although the actual number may be greater orlesser, as well as links 120 x, shown here as links 120A, 120B, 120C,120D, and 120E, although the actual number may be greater or lesser.Links 120 x can transport voice, data, video and/or other trafficbetween nodes 110 x.

Each node 110 x may have a variety of resources, shown in theillustration in exemplary fashion as servers and storage, although manydifferent types of resources may exist.

For all nodes 110 x, and resources within each node, a variety of datais maintained. This includes node capability table 132, node utilizationtable 134, and node cost table 136.

A portion of node capability table 132 is illustrated in simplifiedexemplary fashion in Table 1. Briefly, for each node, information aboutthe capabilities and capacity of the node is present.

TABLE 1 Node Capability Table 132 NODE ELEMENT CONFIGURATION CAPACITYSTATUS Chicago Itanium 8-way with 2 GigaFLOPs On-Line Linux Oracle 9iCluster A Chicago zSeries DB2 1.35 On-Line Enterprise GigaFLOPs ServerChicago EMC Mirrored  16 TB On-Line Symmetrix DMX Atlanta EMC NFS 4.3 TBRoutine Celerra Maintenance

Specifically, the table contains information about nodes, the servers,storage, or other relevant resources, the configuration of thoseelements, raw capacity, and status. Additional information such assoftware or firmware revision levels, security level, costs for use ofthe capacity, cost/pricing models, and so forth may be present as well.

A portion of node utilization table 134 is illustrated in simplifiedexemplary fashion in Table 2.

TABLE 2 Node Utilization Table 134 START END DATE/ DATE/ CAPACITY NODEJOB TIME TIME RESOURCE RESERVED Chicago 33 Apr. 25, 2004, Apr. 28, 2004,Itanium Linux .3 GigaFLOPs 8:43:00 AM 9:15:66 PM Cluster A Chicago 137Apr. 27, 2004, Apr. 28, 2004, Itanium Linux .5 GigaFLOPs 8:43:00 AM9:15:22 PM Cluster A Chicago 22 Apr. 28, 2004, Apr. 28, 2004, EMC .4Terabytes 8:43:00 AM 9:15:30 PM Symmetrix DMX

Specifically, the table contains information about the amount and typeof node capacity reserved by each job, and the start date and time andend date and time of the reservation. Alternative embodiments of thistable and accompanying derivative data are within the scope and spiritof the present invention. For example, the table may be viewed as anaggregate set of data representing aggregate utilization percentagesacross all jobs. Consequently, the table may also be visualized as shownbelow in Table 3.

TABLE 3 Node Utilization Table 134 RE- LINK SOURCE CAPACITY RESERVED(Mb/S) Chicago Itanium Linux Cluster A

Chicago EMC Symmetrix DMX

Atlanta Sun Fire 15K with Solaris 8.0

Atlanta HDS Lightning 9900

An exemplary embodiment of a portion of node cost table 136 is shownbelow in Table 4.

TABLE 4 Node Cost Table 136 NODE RESOURCE COST Chicago Itanium LinuxCluster A $.30/second per GigaFLOP Chicago EMC Symmetrix DMX $.27/secondper TB Atlanta Sun Fire 15K with Solaris 8.0 $.37/second per ServerPower Unit Atlanta HDS Lightning 9900 $.17/second per TB

Specifically, the table contains cost information associated with theuse of the node resource. Although the exemplary table shown aboveillustrates static costs, they may vary, e.g., 27 cents from 9 AM to 5PM ET, and 15 cents otherwise. These costs may also be dynamic, changingthrough the actions of a dynamic pricing system (not shown) which may beused in combination with the present system.

Links 120 x may use any of a number of technologies as are known in theart, e.g., electrical, optical, wireless, free space optics, etc.,leveraging dense wave division multiplexing, SONET, frame relay,asynchronous transfer mode, IP, SNA, fibre channel, Ethernet, ResilientPacket Ring, etc. arranged into any of a number of architectures andtopologies, e.g., point-to-point, ring, dual diverse routed path, mesh,hub, dual hub, counter-rotating rings, etc. Here, the links 120 x areshown as logical point-to-point connections.

For all links 120 x, a variety of data is maintained, including linkcapability database 142, link utilization table 144, and link cost table146.

An exemplary embodiment of a portion of link capability database 142 isshown below in Table 5.

TABLE 5 Link Capability Database 142 SERVICE CHANNEL BANDWIDTH/ LINKPROTOCOL THROUGHPUT (Mb/S) Chicago-Atlanta Gigabit Ethernet 80000Chicago-Atlanta Fibre Channel 80000 Chicago-Los Angeles ESCON 80000Atlanta-Los Angeles ATM 80000

Specifically, the table contains capacity information about linksbetween nodes. E.g., as shown, the link between Chicago and Atlanta hasa number of Gigabit Ethernet connections for a total link capacity of 80Gigabits. Optionally and advantageously, additional information, such asthe fact that the 80 Gb/S link capacity is made up of 32 servicechannels each operating at 2.5 Gb/S (OC48), whether the link is up ordown, whether service channels are up or down, specific vendorinformation, patch panel details, network element configurations,topologies for complexes of network elements, and the like may also bemaintained.

Link utilization table 144 contains data on planned utilization inaccordance with scheduled jobs. An exemplary embodiment of a portion oflink utilization table 144 is shown below in Table 6.

TABLE 6 Link Utilization Table 144 BAND- WIDTH START END PRO- RE- DATE/DATE/ TO- SERVED LINK JOB TIME TIME COL (Mb/S) Chicago- 33 Apr. 25,2004, Apr. 28, 2004, Gigabit 35 Atlanta 8:43:00 AM 9:15:66 PM EthernetChicago- 137 Apr. 27, 2004, Apr. 28, 2004, Fibre 57 Atlanta 8:43:00 AM9:15:22 PM Channel Chicago- 22 Apr. 28, 2004, Apr. 28, 2004, ESCON 23Los 8:43:00 AM 9:15:30 PM Angeles Chicago- 43 Apr. 27, 2004, Apr. 28,2004, ESCON 34 Los 8:43:17 AM 9:15:55 PM Angeles Atlanta- 46 Apr. 27,2004, Apr. 28, 2004, ATM 100 Los 8:43:17 AM 9:15:55 PM Angeles

Specifically, the table contains information about the amount of linkcapacity reserved by each job, the start date and time and end date andtime of the reservation, and the amount of facility bandwidth alreadyreserved by scheduled jobs. Alternative embodiments of this table andaccompanying derivative data are within the scope and spirit of thepresent invention. For example, the table may be viewed as an aggregateset of data representing aggregate utilization percentages across alljobs. Consequently, the table may also be visualized as shown below inTable 7.

TABLE 7 Link Utilization Table 144 LINK PROTOCOL BANDWIDTH RESERVED(Mb/S) Chicago- Atlanta Gigabit Ethernet

Chicago- Atlanta Fibre Channel

Chicago- Los Angeles ESCON

Atlanta- Los Angeles ATM

An exemplary embodiment of a portion of link cost table 146 is shown inTable 8.

TABLE 8 Link Cost Table 146 SERVICE CHANNEL LINK PROTOCOL COSTChicago-Atlanta Gigabit Ethernet $.30/second per Mb/S Chicago-AtlantaFibre Channel $.27/second per Mb/S Chicago-Los Angeles ESCON $.37/secondper Mb/S Atlanta-Los Angeles ATM $.17/second per Mb/S

Specifically, the table contains cost information associated with theuse of the link. Although the exemplary table shown above illustratesstatic costs, they may vary, e.g., 27 cents from 9 AM to 5 PM ET, and 15cents otherwise. These costs may also be dynamic, changing through theactions of a dynamic pricing system (not shown) which may be used incombination with the present system.

Node manager 130 enables access to node capability table 132, nodeutilization table 134, and node cost table 136. This access includesread access, as well as the ability to update the data contained inthese tables. Similarly, link manager 140 enables access to linkcapability database 142, link utilization table 144, and link cost table146. This access also includes read access, as well as the ability toupdate the data contained in these tables.

Jobs are submitted via job submission module 164. This creates an entryin job metadata table 162, comprising metadata concerning the job, suchas a job ID, a priority, capabilities and capacity required by the job,e.g., 12 mainframe class servers running MVS, 3 servers running Linuxand the Apache web server, and 50 Terabytes of enterprise storage.

A job planner 160 interacts with node manager 130 and link manager 140to determine an optimal node to run the job, and then interacts with jobscheduler 150 to schedule the job to run with the selected resources.

Job scheduler 150 then causes the job to be run at the specifiedlocation at the specified time for the specified duration. Prior toexecution, during execution, and after completion, job scheduler 150 mayreport the status of the job to job submission module 164.

FIG. 2 illustrates an embodiment of a job scheduling and executionmethod.

In step 201, a job is submitted. Metadata concerning the job, such aspriority, start time, required completion times, environment/resourcecharacteristics/capabilities/capacity required, access bandwidthcharacteristics, and the like may be provided as part of this metadata,or optionally in certain alternate embodiments, determined from themetadata provided.

In step 202, a set of candidate nodes is determined where the remaining(unallocated) capacity of the capabilities required by the job of thenode is sufficient.

In step 203, it is determined whether the set of candidate nodes isempty. If there are no candidate nodes, the method continues on to step211 where it is reported back that the submitted job cannot be run dueto lack of candidate nodes. However, if there are one or more candidatenodes, the method continues to step 204.

In step 204, a subset of candidate nodes is determined where bandwidthcapabilities of the links feeding the node are determined, possiblyincluding a multiplicity of alternate routes to the node. Methods knownin the art, such as min cut/max flow may be used to determine thesufficiency of bandwidth. This subset of nodes meeting node and linkrequirements, over time, is considered to be the set of valid nodes.

Various start times may be considered in accordance with metadataprovided for the job, based on the potentially variable availability ofcapacity of required node and link capabilities, and the potentiallyvariable job requirements for node and link capabilities.

In step 205, if no valid nodes exist, the method continues on to step211 where it is reported back that the submitted job cannot be run dueto lack of valid nodes. However, if one or more valid nodes do exist,the method continues to step 206.

In step 206, the valid node(s) are examined in turn to determine the oneor more optimal nodes. Optimal nodes are those where, if the job were torun at the node, the “total cost” of running the job is minimized. Inone embodiment, this “total cost” may be determined with respect to thecost of all node and link resources engaged (which may be variably ordynamically priced), and other cost considerations (such as delay costsand so forth).

There may be more than one optimal node, i.e., there may be two or morenodes with the same minimal cost to run the job. This may be so due tosameness of cost drivers, e.g., CPU, storage, software, and bandwidthcosts may be the same. However, even when drivers are different thetotal cost may be the same. E.g., to run the job at node A, the CPU andstorage costs may be low, but the network costs high. At node B, the CPUand storage costs may be high, but the network costs may be low.

In any event, in step 207 one of the one or more optimal nodes isselected. This may be done at random, or based on other factors, such asthe desire to load balance, or a preference for a particular site orsites.

In step 208, the job is scheduled. That is, a firm start time at theselected node is defined in accordance with the analysis done in steps202 and 204, and node and link resources are committed and reserved forthe job. Planned utilization of these resources is incrementedaccordingly.

In step 209, when the start time is reached, the job is actually run atthe selected node with the selected resources, until the job completesat step 210, at which point completion of the job is reported.

Also provided, for the methods described herein, are computer programproducts (such as storage media) storing program instructions forexecution on a computer system having at least one data processingdevice, which instructions when executed by the computer system causethe computer system to perform the methods described herein.

Further provided are systems for performing the methods describedherein, the systems including at least one data processing element.Generally, these data processing elements may be implemented as anyappropriate computer(s) employing technology known by those skilled inthe art to be appropriate to the functions performed. The computer(s)may be implemented using a conventional general purpose computerprogrammed according to the foregoing teachings, as will be apparent tothose skilled in the computer art. Appropriate software can readily beprepared by programmers based on the teachings of the presentdisclosure. Suitable programming languages operating with availableoperating systems may be chosen.

General purpose computers may implement the foregoing methods, in whichthe computer housing may house a CPU (central processing unit), memorysuch as DRAM (dynamic random access memory), ROM (read only memory),EPROM (erasable programmable read only memory), EEPROM (electricallyerasable programmable read only memory), SRAM (static random accessmemory), SDRAM (synchronous dynamic random access memory), and Flash RAM(random access memory), and other special purpose logic devices such asASICs (application specific integrated circuits) or configurable logicdevices such GAL (generic array logic) and reprogrammable FPGAs (fieldprogrammable gate arrays).

Each computer may also include plural input devices (for example,keyboard, microphone, and mouse), and a display controller forcontrolling a monitor. Additionally, the computer may include a floppydisk drive; other removable media devices (for example, compact disc,tape, and removable magneto optical media); and a hard disk or otherfixed high-density media drives, connected using an appropriate devicebus such as a SCSI (small computer system interface) bus, an EnhancedIDE (integrated drive electronics) bus, or an Ultra DMA (direct memoryaccess) bus. The computer may also include a compact disc reader, acompact disc reader/writer unit, or a compact disc jukebox, which may beconnected to the same device bus or to another device bus.

The arrangement provides at least one computer readable medium. Examplesof computer readable media include compact discs, hard disks, floppydisks, tape, magneto optical disks, PROMs (for example, EPROM, EEPROM,Flash EPROM), DRAM, SRAM, SDRAM.

Stored on any one or on a combination of computer readable media issoftware for controlling both the hardware of the computer and forenabling the computer to interact with other elements, to perform thefunctions described above. Such software may include, but is not limitedto, user applications, device drivers, operating systems, developmenttools, and so forth.

Such computer readable media further include a computer program productincluding computer executable code or computer executable instructionsthat, when executed, causes a computer to perform the methods disclosedabove. The computer code may be any interpreted or executable code,including but not limited to scripts, interpreters, dynamic linklibraries, Java classes, complete executable programs, and the like.

From the foregoing, it will be apparent to those skilled in the art thata variety of methods, systems, computer programs on recording media, andthe like, are provided.

The present disclosure supports a system for scheduling at least one jobfor execution on a network of nodes joined by links. The system may have(a) a job planner (160) including (a1) a validity portion configured todetermine the nodes' ability to run the one or more jobs at a given timeand (a2) a cost portion configured to determine costs to run the atleast one job at respective nodes at a given time. The system may alsohave (b) a job scheduler (150) configured to cooperate with the validityportion and the cost portion so as to determine at least one optimumnode that is able to run the at least one job and that has a lowestcost.

The system may also have a job metadata storage portion (162) configuredto store job metadata for the at least one job; a node characteristicsstorage portion (132, 134) configured to store node characteristics; anda link characteristics storage portion (142, 144) configured to storelink characteristics. The validity portion may be configured todetermine a possible validity of a given node by comparing the jobmetadata with the node characteristics of the given node and the linkcharacteristics.

The system may also have a node cost storage portion (136) configured tostore node-specific cost data; and a link cost storage portion (146)configured to store link-specific cost data for combinations of jobmetadata, node(s), and link(s). The cost portion may be configured todetermine a total cost based on the job metadata, the node-specific costdata, and the link-specific cost data.

The link-specific cost data (146) may include at least one from a groupincluding at least one rate dependent on volume of bandwidth; at leastone rate dependent on volume of data transported; at least one ratedependent on level of jitter; at least one rate dependent on droppedpackets; and at least one rate dependent on network latency.

The node-specific cost data (136) may include at least one from a groupincluding at least one rate dependent on processor utilization; and atleast one rate dependent on storage utilization.

The job metadata (162) may include at least one cost function dependenton completion delay.

The system may also have a portion (150) configured to cause the atleast one job to be run at the one or more nodes that are able to runthe job at the lowest cost.

Each job may be associated with a job transport capacity requirement;all the costs may be set to zero so that the job scheduler (150)cooperates with the validity portion so as to determine at least onenode that is able to run the at least one job; and the system may alsohave a link manager (140) configured to reserve at least some of thelink capacity of at least one of the links connected to the one or moreof the nodes where the job is scheduled, such that a resulting reservedlink capacity matches or exceeds the job transport capacity requirement.

The nodes may include components having at least a capability and apredetermined capacity; the links may include channels having at least acapability and a predetermined capacity; and the system may also have ajob schedule (150) configured to store a list of jobs to be run, andplanned nodes and links to be at least partially utilized by each jobover time; a job metadata storage portion (162) including, for each job,component capability and capacity requirements, link capability andcapacity requirements, and job duration and deadline; a job submissionportion (164) configured to receive the metadata; a node capabilitydatabase (132) configured to maintain information about the capabilityand capacity of each component within the nodes; a node utilizationtable (134) configured to maintain information about a plannedutilization of capacity of each component of the nodes in accordancewith the job schedule (150); a node cost table (136) configured tomaintain information on a cost of the capacity of each component at thenodes over time; a link capability database (142) configured to maintaininformation about a capability and capacity of each channel within thelinks; a link utilization table (144) configured to maintain informationabout a planned utilization of capacity of each channel of the one ormore links in accordance with the job schedule (152); a link cost table(146) configured to maintain information on a cost of the capacity ofeach channel of the links over time; a node manager (130) configured tomanage the node capability database (132), the node utilization table(134), and the node cost table (136), and to modify the node utilizationtable; a link manager (140) configured to manage the link capabilitydatabase (142), the link utilization table (144), and the link costtable (146), and to modify the link utilization table; a job planner(160) that, when provided a new job and its associated metadata, isconfigured to communicate with the link manager (140) and the nodemanager (130) to determine whether there is at least one validcombination of nodes and links with the capability and capacity overtime to complete the at least one job by the deadline; a job planner(160) that may include a cost module configured to select an optimaltotal cost combination of nodes and links from among the at least onevalid combination of nodes and links with the capability and capacityover time to complete the job by the deadline; and a job scheduler (150)that, if there is at least one valid combination of node and links withthe capability and capacity over time to complete the at least one jobby the deadline; is configured to schedule the at least one job inaccordance with the selected optimal total cost combination, and tocommunicate with the node manager (130) and link manager (140) to updatethe link utilization table (144) and the node utilization table (134) inaccordance with the selected optimal total cost combination, and toupdate the job schedule (152), and to cause the at least one job toexecute in accordance with the selected optimal total cost combinationof nodes and links.

The present disclosure also supports a system for scheduling at leastone job for execution on a network of nodes joined by links havingrespective link capacities, each job associated with a transportcapacity requirement. The system may have a job scheduler (150)configured to schedule the at least one job to be executed on at leastone selected node; and a link manager (140) configured to reserve atleast some of the link capacity of at least one of the links connectedto the at least one selected node, to match the job transport capacityrequirement.

The system may also have a job planner (160) including a validityportion configured to determine the nodes' ability to run the one ormore jobs at a given time; and the job scheduler (150) may be furtherconfigured to cooperate with the validity portion so as to determine theat least one selected node.

The system may also have a job planner (160) including a cost portionconfigured to determine costs to run the at least one job at respectivenodes at a given time; and the job scheduler (150) may be furtherconfigured to cooperate with the cost portion so as to determine the atleast one selected node to have a lowest cost.

The present disclosure further supports a method for scheduling at leastone job for execution on at least one optimum node in a network of nodesconnected by links. the method may involve (202) determining which nodesare capable of running the at least one job; (206) analyzing measures oftotal cost associated with execution of the at least one job onrespective nodes that were determined to be capable of running the atleast one job; and (207) based on a measure of least total cost,selecting the at least one optimum node for running the at least onejob.

The method may also involve (208) scheduling the at least one job forexecution; and (209) causing the at least one job to be run at the atleast one optimum node.

The step (206) of analyzing measures of total cost, may include at leastone from a group including: evaluating at least one rate dependent onvolume of bandwidth; evaluating at least one rate dependent on volume ofdata transported; evaluating at least one rate dependent on level ofjitter; evaluating at least one rate dependent on dropped packets; andevaluating at least one rate dependent on network latency.

The step (206) of analyzing measures of total cost, may include at leastone from a group including: evaluating at least one rate dependent onprocessor utilization; and evaluating at least one rate dependent onstorage utilization.

The step (206) of analyzing measures of total cost, may includeevaluating at least one cost function dependent on completion delay.

Step (202) of determining which nodes are capable of running the atleast one job may include determining a planned utilization of the linksover time, limited by a capability and a capacity of the respectivelinks; determining a planned utilization of the nodes over time, limitedby a capability and a capacity of the respective nodes; determining zeroor more candidate nodes having a capability and a capacity that meets orexceeds specified requirements of the at least one job; for eachcandidate node, determining link requirements over time on the links ifthe at least one job were run at that candidate node; determiningwhether the link requirements for each candidate node can be met, giventhe planned utilization of the links over time; and determining zero ormore valid nodes to be those candidate nodes that allow linkrequirements to be met, while running the at least one job.

The step (207) of selecting the at least one optimum node, may includedetermining a total cost to run the job for each valid node; andselecting the at least one optimum node such that there is no othervalid node with a lower total cost.

The method may also involve selecting a planned node from among the atleast one optimum node; reserving at least a portion of a remainingcapability and capacity of each of the links that are required to meetthe specified requirements of the at least one job planned to run at theplanned node; reserving at least a portion of a remaining capability andcapacity of the planned node required to meet the requirements of thejob; and causing the job to be executed at the planned node.

The present disclosure further supports a computer program productincluding computer executable code or computer executable instructionsthat, when executed, causes a at least one computer to perform themethods described herein.

The present disclosure further supports a system configured to performthe methods described herein.

Many alternatives, modifications, and variations will be apparent tothose skilled in the art in light of the above teachings. While thepresent invention has been described with reference to one or moreparticular embodiments, those skilled in the art will recognize thatmany changes may be made thereto without departing from the spirit andscope of the present invention. For example:

-   -   The system may be implemented with elements organized and        functioning in a way other than that specifically illustrated in        FIG. 1A. For example, the job planner 160 and job scheduler 150        may be integrated into a single module; conversely, functions        performed within a single block may be decomposed into plural        blocks. Thus, the performance of the various functions may be        distributed among elements differently from the examples        disclosed above.    -   Costs may be ignored, but bandwidth reserved on demand, simply        by setting all costs to zero. This may be done explicitly, or        implicitly by removing the cost tables and costing steps from        the system and method respectively. In any event, optimum nodes        may or may not be determined while the bandwidth on demand (BoD)        function is performed.    -   Jobs may be viewed as potentially resident at more than one node        simultaneously (distributed application architecture, disk        mirroring or tape backup and restore, etc.), wherein the system        components and method steps would consider, in an alternate        embodiment, various multi-node solutions.    -   Rather than explicitly considering each node, a sample of nodes,        or those with the lowest utilization, or lowest cost, or        combination of lowest cost or utilization may be considered,        thus reducing the search time.    -   As jobs run, actual information may be used to upgrade forecast        capacity requirements, and the job schedule updated accordingly.    -   Lower priority jobs may be dropped from the job schedule or        delayed in accordance with a policy, to maximize the number of        higher priority jobs successfully run.    -   Rather than considering node capability and capacity first and        then link capability and capacity, the order may be reversed.    -   Rather than first generating a set of candidate nodes, and then        reducing that set down to a set of valid nodes, each node may be        considered in turn. If this is done, a minimal cost objective        may be set which if a valid configuration is found meeting the        cost objective, that configuration would be selected as        satisfactory, if not in fact optimal.    -   Other embodiments and variations not specifically listed.

Each of these embodiments and variations thereof are contemplated asfalling within the spirit and scope of the claimed invention, which isset forth in the following claims. It is therefore to be understood thatwithin the scope of the appended claims and their equivalents, theinvention may be practiced otherwise than as specifically describedherein.

The invention claimed is:
 1. A machine readable storage devicecomprising instructions which, when executed, cause a machine to performoperations comprising: selecting a total cost combination of nodes andlinks from among valid combinations of nodes and links from a networkwith capability and capacity over time to complete a computer-executablejob by a deadline; setting costs associated with different combinationsof the nodes and the links from the network of nodes to zero;determining whether the combinations of the nodes and the links with thecapability and the capacity over time to complete thecomputer-executable job by the deadline are valid based on determining apossible validity of a given node by comparing job information with:stored node characteristics of the given node, and stored linkcharacteristics of the links associated with the give node; scheduling afirst node for the selected combination of the nodes and the links toexecute the computer-executable job, the scheduling based on binarymachine code comprising the computer executable job; and reserving linkcapacity of a link connected to the first node, to match a job transportcapacity requirement of the computer-executable job.
 2. The machinereadable storage device of claim 1, wherein the scheduling of the firstnode is based on respective abilities of the nodes from the selectedcombination of the nodes and the links to execute thecomputer-executable job at a given time.
 3. The machine readable storagedevice of claim 1, wherein the operations further comprise determiningcosts to execute the computer-executable job at respective nodes in thecombinations of the nodes and the links at a given time, and wherein thescheduling of the first node is based on the first node having a lowestcost among the respective nodes.
 4. The machine readable storage deviceof claim 1, wherein selecting a total cost combination of the nodes andthe links further comprises: storing node-specific cost data in a nodecost storage; storing link-specific cost data for combinations of jobmetadata, nodes, and links in a link cost storage; and determining atotal cost corresponding to the total cost combination of nodes andlinks based on the job information, the node-specific cost data, and thelink-specific cost data.
 5. The machine readable storage device of claim4, wherein the link specific cost data includes at least one of: a ratedependent on volume of bandwidth; a rate dependent on volume of datatransported; a rate dependent on level of jitter; a rate dependent ondropped packets; and a rate dependent on network latency.
 6. The machinereadable storage device of claim 4, wherein the node specific cost dataincludes at least one of: a rate dependent on processor utilization; anda rate dependent on storage utilization.
 7. The machine readable storagedevice of claim 1, wherein the job information includes a cost functiondependent on completion delay.
 8. A machine readable storage devicecomprising instructions which, when executed, cause a machine to performoperations comprising: determining which nodes from a network of nodesconnected by links are capable of executing a computer-executable jobby: determining a planned utilization of the links over time, limited bya capability and a capacity of the respective links; determining aplanned utilization of the nodes over time, limited by a capability anda capacity of the respective nodes; determining zero or more candidatenodes having a capability and a capacity that meets or exceeds specifiedrequirements of the computer-executable job; for each of the candidatenodes, determining link requirements over time on the links if thecomputer-executable job were executed at that candidate node;determining whether the link requirements for the candidate nodes can bemet, given the planned utilization of the links over time; anddetermining zero or more candidate nodes as valid nodes that allow linkrequirements to be met, while executing the computer-executable job;analyzing measures of total cost associated with execution of thecomputer-executable job on respective one of the nodes that weredetermined to be capable of executing the computer-executable job; basedon a measure of least total cost associated with the execution of thecomputer executable job on the respective nodes, selecting a selectednode from the network of nodes to execute the computer-executable job,the selection based on object code comprising the computer-executablejob; scheduling the computer-executable job for execution; and causingthe computer-executable job to be executed at the selected node.
 9. Themachine readable storage device of claim 8, wherein the analyzing of themeasures of total cost, includes at least one of: evaluating a ratedependent on volume of bandwidth; evaluating a rate dependent on volumeof data transported; evaluating a rate dependent on level of jitter;evaluating a rate dependent on dropped packets; and evaluating a ratedependent on network latency.
 10. The machine readable storage device ofclaim 8, wherein the analyzing of the measures of total cost, includesat least one of: evaluating a rate dependent on processor utilization;and evaluating a rate dependent on storage utilization.
 11. The machinereadable storage device of claim 8, wherein the analyzing of themeasures of total cost, includes evaluating a cost function dependent oncompletion delay.
 12. The machine readable storage device of claim 8,wherein the selecting of the selected node includes: determining a totalcost to execute the computer-executable job for each valid node; andselecting the selected node such that there is no other valid node witha lower total cost.
 13. The machine readable storage device of claim 12,wherein the selected nodes comprise at least two selected nodes and theoperations further comprise: selecting a planned node from among the atleast two selected nodes; reserving at least a portion of a remainingcapability and capacity of each of the links that are required to meetthe specified requirements of the computer-executable job planned to beexecuted at the planned node; reserving at least a portion of aremaining capability and capacity of the planned node required to meetthe requirements of the computer-executable job; and causing thecomputer-executable job to be executed at the planned node.
 14. Amachine readable storage device comprising instructions which, whenexecuted, cause a machine to perform operations comprising: determiningabilities of respective nodes to execute a computer-executable job at agiven time by: determining a planned utilization of the respective nodesover time, limited by a capability and a capacity of the respectivenodes; and determining zero or more candidate nodes having a capabilityand a capacity that satisfies specified requirements of thecomputer-executable job; determining whether there is a validcombination of the nodes and links with capability and capacity overtime to complete the computer-executable job by a deadline based ondetermining a possible validity of a given candidate node by comparingjob information with node characteristics of the candidate node and linkcharacteristics of the links associated with the candidate node;determining costs to execute the computer-executable job at respectiveones of the nodes in the valid combination at the given time; selectinga total cost combination of the nodes and the links from among the validcombination; and scheduling the computer-executable job to execute on afirst one of the nodes from the valid combination that has a lowest costto execute the computer-executable job at the given time, the schedulingof the computer-executable job based on object code comprising thecomputer-executable job.